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ABSTRACT 



Are states setting high expectations for student learning? 

To answer this question the Council for Basic Education (CBE) evaluated the 
rigor of mathematics and language arts standards in the United States. This 
report is divided into the following sections: (1) Context for the Study; (2) 

Rigor in State Standards; (3) Issues Affecting the Evaluation of Rigor in 
State Standards; (4) Application of the Evaluation Process; (5) Process 
Design; and (6) Future Directions. Evaluation of the standards documents from 
43 states yields several important findings: (1) in language arts, seven 

states had very rigorous standards, 21 had rigorous standards, and 14 had 
standards with low levels of rigor; (2) in mathematics, 16 states had very 
rigorous standards, 24 had rigorous standards, and three had standards with 
low levels of rigor; (3) state mathematics standards tend to be more rigorous 
than language arts standards; (4) most states' language arts standards 
address basic skills but do not address literature study, research, or 
language study such as word origins and differences between standard usage 
and slang; (5) most states' mathematics standards contain few major gaps in 
the concepts or skills included, and states with low to moderate levels of 
rigor tend to address the most essential concepts and skills but at a lower 
level of sophistication; and (6) states are incorporating both concepts 
(e.g., algebra and geometry) and skills (e.g.# problem solving and reasoning) 
into their standards. Appendices contain advisory panels, grades for rigor of 
state standards, frameworks for mathematics and language arts, and scoring 
rubrics. (PVD) 
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Foreword 



P oll after poll shows that students, parents, teachers, administrators, business leaders, and the public 
at large support setting higher, or more rigorous, academic standards. But are the standards set by 
the states in core academic subjects, such as mathematics, English language arts, science, and 
history/social studies, so low now? And what does it mean to have rigorous standards? Until now, no one 
has answered these essential questions in a well-defined, systematic way, yet CBE has always been 
concerned with these questions. Our 1956 charter speaks of the importance of high standards for all 
children in the basic liberal arts, and for 42 years we have pursued that mission with passion. 

Therefore, when Ron Wolk, the then-publisher of Education Week, talked with me about the need to design 
and apply an analysis of the rigor of state standards, I was eager but a bit hesitant to have CBE accept this 
assignment. To do so, CBE would have to be a pioneer. To the best of our knowledge, we are the first orga- 
nization to undertake a systematic, analytical review to determine the rigor of state standards. 

What follows in the pages of this Special Report is a discussion of our study to rate states for their efforts 
to set high standards. Others, such as the American Federation of Teachers (AFT), have rated states’ stan- 
dards for clarity and specificity. CBE has built upon that work by adding the dimension of rigor to 
determining the overall quality of state standards. 

We make no pretense that what we have done is perfect, for it is not. We are all learning, and we seek to 
learn much from reaction to this report. We have sent individual state findings to each state’s superintendent 
and school board president, so that they may use our results to improve their state standards. We plan to 
expand our next study to include assessments of additional subject areas. 

I wish to acknowledge the many contributions of those who assisted in this study, the advisory panelists, 
the staff at state departments of education, the reporters at Quality Counts '98: The Urban Challenge, and 
the members of the evaluation team. I also thank the CBE staff who made this work possible: Kaye 
McCann, who directed the project; Scott Joftus and Ilene Berman, who led the evaluation teams and 
authored the technical report; Derise Wright and Daniel Iny, who provided administrative assistance; and 
Madelyn Holmes, who served as the report editor. I am further grateful to the supporters of this analysis 
and report: Circuit City Foundation, Leon Foundation, Motorola Foundation, State Farm Insurance 
Companies, and the United States Department of Education, Planning and Evaluation Service. 

With this study, CBE contributes to the vibrant and critical discussion of standards and provides footholds 
for all of us striving to ensure that every student receives a high-quality education. 



Christopher T, Cross 

President 

Council for Basic Education 
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Executive Summary 



Are states setting high expectations for student learning? To answer this question, Education Week asked 
the Council for Basic Education (CBE) to evaluate the rigor of states’ mathematics and English language 
arts standards. Next year, CBE will build on this analysis by evaluating the rigor of states’ science and 
social studies standards, in addition to their mathematics and English language arts standards. 

All states except two -- Iowa and Wyoming — have either developed or are in the process of developing 
standards. Of those states still in the process of developing standards, five are not far enough along in 
mathematics and six are not far enough along in English language arts for CBE to evaluate. This leaves 43 
states with mathematics standards and 42 states with English language arts standards ready for CBE to 
evaluate. CBE’s evaluation of these standards documents resulted in several important findings: 

• In English language arts, seven states were found to have very rigorous standards; 21 states were found 
to have rigorous standards; and 14 states were found to have standards with low levels of rigor. 

• In mathematics, 16 states were found to have very rigorous standards; 24 were found to have rigorous 
standards; and three states were found to have standards with low levels of rigor. 

• State mathematics standards tend to be more rigorous than English language arts standards. 

• Most states’ English language arts standards address basic skills such as reading, writing, and using 
correct grammar, punctuation, and spelling. 

• Many states’ English language arts standards do not address: 

■ specific reading requirements (e.g., how much and what types of materials students 
are expected to read); 

■ literature study (e.g., reading literature of particular time periods and genres); 

■ research (e.g., giving precise credit for others’ ideas and gathering information from a 
variety of sources); and 

■ language study (e.g., examining word origins and recognizing differences between standard 
English usage and slang). 

• Most states’ mathematics standards contained few major gaps in the concepts or skills they included. 
States with low to moderate levels of rigor in their mathematics standards tend to address most essential 
concepts and skills, although at a lower level of sophistication than states with very rigorous standards. 

• States are wisely incorporating both concepts (e.g., algebra and geometry) and skills (e.g., problem 
solving and reasoning) in their standards, despite a sometimes damaging national debate over whether to 
exclude one or the other. 

To arrive at these findings, CBE developed an evaluation process that is analytical, concrete, and clear, and 
therefore replicable. The development and implementation of the process were guided by two advisory 
panels -- one for mathematics and one for English language arts - comprised of subject specialists, teach- 
ers, parents, and business representatives. The panelists helped to develop CBE’s definition of rigor in 
standards. The panelists also improved upon and approved the frameworks and rubrics CBE created to 
evaluate the rigor of state standards. 
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After the panels’ review, CBE trained evaluators to use the definition, frameworks, and rubrics, and to ana- 
lyze the standards. The panelists reviewed a sample of the evaluators’ work, and CBE staff made changes 
to the analytical process when appropriate. Once final changes to the process were made, the evaluators, 
all former or current teachers in the subject they evaluated, analyzed the state standards. CBE sent the 
final, detailed analysis to each state superintendent and state board of education president. 

This report — the culmination of the first ever analytical assessment of the rigor of state standards — 
includes a great deal of information that should be of help and interest to educators, parents, policy makers, 
media representatives, and business leaders. In particular, the report addresses the following questions: 

• What are standards, and why are they important? 

• What is rigor, and why is it important? 

• Are states’ mathematics and English language arts standards rigorous? 

• What are the strengths and weaknesses of states’ mathematics and English language arts standards? 

• What was the process used to evaluate the rigor of state standards? 

• What issues arose during the evaluation of the rigor of state standards? 

• What are future plans for expanding upon this analysis? 



CBE’s study, like the development of state standards, is a work in progress. Just as states should continue 
to improve their standards and standards-driven systems (i.e., assessments, accountability, curriculum, 
professional development, technology), CBE will continue to improve our understanding of rigor and 
related criteria for high-quality standards, such as clarity, specificity, and measurability. 
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I. Context for the Study 

Introduction 

A foundational tenet of American society is that all children should have access to and be afforded a good 
education. To this end, most states across the nation have established academic standards to define what 
students need to know and be able to do at key points in their educational careers. Ideally, these standards 
should set high expectations in order to challenge students to achieve to the best of their abilities. In reality, 
states have developed standards that range widely in their rigor, meaning that there are different expecta- 
tions for student learning based solely upon where students live. Until now, however, there has been no 
systematic way to assess the rigor of the expectations states have set for students. The Council for Basic 
Education (CBE) addressed this need by first developing an operational definition of rigor and then applying 
that definition to evaluate the rigor of state standards in mathematics and English language arts. 

This study was initiated when Education Week asked CBE to assess the rigor of state standards for its 
second annual report card on the condition of public education in each state, entitled Quality Counts *98: 
The Urban Challenge. Last year in its first edition. Quality Counts *97 evaluated all states on many impor- 
tant education indicators, including standards. This evaluation, however, was based solely on the existence 
of standards and not the rigor of those standards. 

CBE has supplemented Education Week*% overall assessment of the states’ education systems by conducting 
an analysis of the rigor of state English language arts and mathematics standards. In particular, our study 
looked at standards written for grades 4 and 12 in English language arts and grades 8 and 12 in mathe- 
matics. Research shows that students unable to read well by the end of the third grade are more likely to 
struggle in later grades or to drop out, and will have fewer good job options when they enter the work 
force. Similarly, developing mathematics proficiency by eighth grade enables students to succeed in high 
level math and science courses in high school, which boost scores on college entrance exams and prepare 
students for the intellectual challenges of college and careers. Our decision to examine standards written 
for grade 4 in English language arts and grade 8 in mathematics was also guided by the fact that these are 
the grades at which President Clinton’s proposed voluntary national tests in reading and mathematics are 
to be administered. In 1999, CBE will expand the scope of our work to include assessments of rigor in his- 
tory/social studies and science as well as mathematics and English language arts and will make modifica- 
tions to our process as warranted. 

A Brief Discussion of Standards 

High standards are a national priority. 

The establishment of “world-class” standards was one of six goals for the U.S. education system set by 
President Bush and the nation’s governors in 1989. In 1994, President Clinton signed the Goals 2000: 
Educate America Act, which provides states with some federal funding to help them establish internationally 
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competitive standards for their students. In 1996, the governors from nearly every state and some of the 
nation’s top CEOs met to discuss the need for higher academic standards.' Though they vary in overall 
quality, standards have been adopted by or are under development in 48 states. 

Students^ educators, and communities benefit from standards. 

Standards are statements that express what students should know and be able to do by specified points in 
their educational careers. Standards provide at least four benefits. First, standards set clear, high expecta- 
tions for student achievement and communicate these expectations to parents, teachers, students, and the 
public. Second, standards provide a basis for holding educators and students accountable. If students fail 
to meet standards, remedial actions can be taken. For example, students can be required to attend summer 
school, and schools with a large number of students failing to meet standards can be required to develop 
and implement a sound plan for improving school academic programs. 

A third benefit of standards is that they promote educational equity by demanding that all students achieve 
at high levels. Historically, many educators have assumed that some students — especially poor and minority 
ones — were not capable of learning essential, high-level concepts and skills. Consequently, schools have 
failed to offer a significant number of these students opportunities for learning at high levels and rich 
curricular experiences. Finally, by stating explicitly the goals for student learning, standards help to guide 
efforts to measure student achievement, improve teacher training, develop effective curricula and instruc- 
tional strategies, and allocate resources where they are most needed. 

Rigorous standards allow for strong standards-driven reform. 

States with rigorous standards have a solid foundation for creating an entire standards-driven system, 
assuming that the standards are also clear, widely distributed, and supported by the public. If a state’s stan- 
dards are appropriately high, they serve as a solid base for creating other essential components of the sys- 
tem, including assessments, which determine whether students are meeting the standards; curricula, which 
provide teachers with guidance for developing lessons linked to the standards; professional development, 
which helps teachers to teach students more effectively; resource allocation, which determines program 
funding; and accountability systems, which impose consequences (including assistance and disciplinary 
action) for students and schools that do not meet expectations. 

Students in states with rigorous standards should score higher on assessments than students in states with 
weak standards and should be better prepared for post-secondary education, careers, and lifelong learning. 
This belief assumes that teachers implement the standards, that the standards are aligned to assessments, 
and that students and schools are held accountable for student achievement. Although rigorous standards 
by themselves do not guarantee student achievement, they are an essential first step toward that end. 

CBE will use the term standard consistently in this report. 

The word standard means many things to many people. States use various terms to describe their expec- 
tations for student learning, including curriculum frameworks, goals, learning outcomes, proficiencies, and 
benchmarks. In this report, state standards documents and individual standards statements within these 
documents will be referred to as state standards. CBE adapted model rigorous standards against which all 
state standards were compared. These model standards, 81 for mathematics and 62 for English language 
arts, will be referred to as CBE's framework benchmarks. 



'Throughout this document, the term standard will be used to refer to academic standards. 
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The Call for High Standards 

Previous studies have begun to assess the quality of standards. 

Public support for academic standards and the recognition of the need for “world-class,” “high,” or 
“rigorous” standards have steadily increased since 1989, when the first education summit was held in 
Charlottesville, Virginia and the National Council of Teachers of Mathematics (NCTM) published the first 
standards document. Though there is widespread agreement on the need for rigor, there has not been agree- 
ment on what constitutes rigor. Complicating matters is the fact that rigor is only one, albeit the most 
important, component of high-quality standards. In addition to being rigorous, high-quality standards 
should be clear, specific, measurable, and linked to assessments and curricula. 

Several organizations have attempted to define high-quality standards. The Council for Basic Education, 
the Mid-continent Regional Laboratory, the Council for Chief State School Officers, the Thomas B. 
Fordham Foundation, and the American Federation of Teachers (AFT) among others have either published 
reports that discuss the criteria for high-quality standards, worked directly with states to create or evalu- 
ate state standards, or both. The AFT’s Making Standards Matter 1997, the most widely cited evaluation 
of state standards, has assessed the clarity and specificity of states’ English, mathematics, science, and 
social studies standards. 

The AFT has called upon states to make sure that their standards are rigorous but acknowledges that “this 
is difficult to accomplish without a reliable set of resources and a tried-and-true technology for bench- 
marking academic standards” and that “states need help to make sure that their standards are rigorous and 
internationally competitive.”^ CBE has heeded AFT’s call by developing, applying, and disseminating 
information about how to benchmark state standards. 

Deflning Rigor in Standards 

Most people seem to have an intuitive sense of what constitutes a rigorous education. This sense often 
includes the idea of student mastery of advanced topics in traditional disciplines, such as English language 
arts, civics, geography, history, mathematics, science, the arts, and foreign languages. Standards should 
capture this sense of rigor and should clearly and simply spell out the topics that all students should master. 

Still, there remains the question: What topics, and at what level of depth, should a rigorous set of standards 
include? Those concerned with the quality of education struggle to answer this question without having a 
set of standards as a starting point. Once afforded the opportunity to review a set of standards, however, 
many people will pass judgement on the standards’ rigor, claiming that certain topics were omitted or that 
student learning was required at a level too high or too low. This conundrum brings to mind a famous case 
tried before the U.S. Supreme Court, in which former Justice and CBE board member Potter Stewart 
asserted that, although he could not settle on a definition of pornography that could be universally applied, 
he knew it when he saw it. 

CBE decided that when it comes to rigor in standards, simple recognition is not enough. States and 
districts need assistance in defining rigor so that they may set expectations that are high, yet appropriate 
for all students. Until now, though, there has not been a definition of rigor that could be used to conduct a 
reliable, objective, case-by-case assessment of the rigor of state standards. Defining rigor in standards is 
challenging, because not everyone agrees upon what is most important for students to learn or the level at 
which all students can achieve. 



^AFT Press Release, “AFT Report Shows Quality of State Academic Standards Improving, Stakes for Students Still a 
Problem,” July 29, 1997. 
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CBE decided that the benefits of defining rigor far outweigh the inherent difficulties, and that these 
acknowledged drawbacks could be minimized, given a conscientious definition and analysis process. 

Rigor is a critical element of high-quality standards. 

CBE’s analysis was designed solely to examine the rigor of state standards, yet we acknowledge that many 
other factors determine the overall quality of standards. High-quality standards set expectations for stu- 
dents to learn important, advanced academic content and to acquire and exercise skills (e.g., problem solving, 
communication) in accordance with their developmental readiness. High-quality standards convey these 
expectations clearly; they are well organized and written in language that everyone can understand. High- 
quality standards are written at a level of specificity detailed enough to provide teachers with guidance for 
planning tests and lessons without dictating instructional practice. High-quality standards compare favor- 
ably with the best standards of the nation and the world and lay a solid foundation for a coordinated system 
that may also include curricula, assessments, professional development, and accountability systems. 

An evaluation of rigor acknowledges clarity, organization, and specificity. 

During our analysis of the rigor of standards, we did not pass judgement on the clarity, organization, or 
specificity of the state standards documents. Attempting to differentiate these characteristics from one 
another, however, proved difficult. Though we were not evaluating clarity, unclear language hampered our 
ability to understand a document’s intentions. Judging organization was not in the purview of our study 
either, but having to shuffle through pages of a state’s standards document in search of language that could 
be compared to CBE’s framework benchmark might have caused us to overlook important material. While 
we were not concerned with grading the specificity of a document, some level of specificity is inherent in 
our framework benchmarks, and thus necessary in the state standards to determine alignment between the 
two. Figure 1 presents the definitions of rigor CBE developed in consultation with content advisory pan- 
els and used to evaluate state mathematics and English language arts standards. 

Figure 1 

CBE's Definition of Rigor in Mathematics and English Language Arts Standards 
Rigor in Mathematics 

The standards should require all students, at the appropriate grade level, to learn the 
essential concepts and skills of mathematics at the level of sophistication specified by 
the National Assessment of Educational Progress (NAEP) in mathematics and the 
National Council of Teachers of Mathematics (NCTM).’ 

Rigor in English Language Arts 

The standards should require all students, at the appropriate grade level, to learn the 
essential concepts and skills of English language arts at the level of sophistication specified 
by the Council for Basic Education's Standards for Excellence in Education (SEE), which 
were written in consultation with subject experts who drew inspiration from exemplary 
state and national standards documents.* 



CBE’s definition of rigor incorporates essential and sophisticated learning. 

Both definitions of rigor incorporate two important ideas. To be rigorous, standards must address essential 
concepts and skills. Standards must also require student understanding and application of these essential 

^ NAEP and NCTM will be described in detail in Section V. 

^ SEE will be described in detail in Section V, 
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concepts and skills at a level of sophistication or complexity that is appropriate and challenging to students 
at a particular grade level. The graph in Figure 2 depicts how the relationship between the inclusion of essen- 
tial concepts and skills and the presentation of these concepts and skills at appropriately high levels of 
sophistication determine a state’s standards’ overall grade for rigor. If state standards lack either compo- 
nent, the grade for rigor decreases accordingly. 

Figure 2 

A Schematic Definition: 

Rigor in Standards is Determined by Inclusion 
and Sophistication of Essential Concepts and Skills 



An example will help to illustrate this point. A mathematics standard for twelfth grade might require 
students to know how to solve equations. While solving equations is an essential concept, this standard is 
not written at a high level of sophistication, because an equation can be an extremely easy concept (for 
example, 2+3=5). To make this standard appropriately sophisticated for twelfth graders, the standard might 
require students to solve quadratic equations (which take the form of ax^+bx+c=y) with one term unknown 
(e.g., 3x^+4x+5=25). The standard could be even more sophisticated, if it required students to explain how 
a quadratic equation may be applied to model and solve a real world situation. 

CBE’s models of rigor have been approved by many. 

CBE’s definition of rigor was developed with the assistance of and approved by expert panels convened 
by CBE for this study (see Appendix A for a list of panel members; see Section V for more information 
about the panels and for a discussion of the project design). Panels also aided in the development and 
approval of CBE’s frameworks for both mathematics and English language arts, which were used as the 
basis for comparing and evaluating all state standards. The frameworks are adapted from highly regarded, 
well-established documents that were developed and approved by experts. Though CBE’s definitions and 
frameworks may not enjoy universal acceptance, we believe that we have created the strongest possible 
foundation for evaluating the rigor of states’ mathematics and English language arts standards. 
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II. Rigor in State Standards 



For this study, CBE reviewed state board of education-approved mathematics and English language arts 
standards documents.^ CBE also reviewed some draft standards documents. Drafts were reviewed only 
when the state board of education assured CBE that these draft standards would be approved without sig- 
nificant changes by January 1, 1998. Figure 3 shows the states that have either no standards, standards 
under development, or drafts of standards ready for review. All states not listed in Figure 3 have standards 
that have been approved by their state boards. 

Figure 3 



Status of State Standards Development 





No standards 


Standards under 
development, not 
ready for review 


Standards expected to 
be approved by the 
state board of education 
by January 1998 


Mathematics 


Iowa 

Wyoming 


California 
Idaho 
Nevada 
South Dakota 
Tennessee 


Connecticut 

Georgia 

Pennsylvania 

Wisconsin 


English 

Language Arts 


Iowa 

Wyoming 


Connecticut 

Georgia 

Idaho 

Nebraska 

Nevada 

South Dakota 


California 

Pennsylvania 

Wisconsin 



CBE evaluated the mathematics standards of 43 states and the English language arts standards of 42 states 
(see Appendix B for the grades for rigor of all the states’ mathematics and English language arts standards). 
To do so, CBE compared the states’ standards to CBE’s framework benchmarks (81 for mathematics and 
62 for English language arts). The mathematics benchmarks were written at the eighth and twelfth grade 
levels, and the English language arts benchmarks were written at the fourth and twelfth grade levels. 

For each benchmark, states received a score of zero, one, two, three, or four, depending on how the rigor 
of their standards compared to the rigor in the framework benchmarks. Standards with an unacceptably 
low level of rigor were given scores of zero or one. Standards with an acceptable level of rigor were given 
a score of two, and standards with high levels of rigor were given scores of three or four. Scores corre- 
sponding to each framework benchmark were averaged and converted into a final grade that represents 
CBE’s overall assessment of the rigor of a state’s standards. Standards with a grade of B+ or higher are 
considered very rigorous; standards with a grade between C and B are considered rigorous; and standards 



^ Wisconsin has no state board of education, only a state department of education. 
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with a grade lower than C are considered to contain low levels of rigor. Section V describes in detail the 
process of evaluating the states’ standards used by CBE. 



Mathematics standards are more rigorous than English language arts standards. 

As shown in Figure 4, only seven percent of the states’ mathematics standards were found to lack rigor, 
whereas 33 percent of the states’ English language arts standards were found to contain low levels of rigor. 



Figure 4 

Percentage of States with Very Rigorous, 
Rigorous, and Low Levels of Rigor in English 
Language Arts Standards 




Percentage of States with Very Rigorous, 
Rigorous, and Low Levels of Rigor in 
Mathematics Standards 



7% 




■ Very Rigorous 
^ Rigorous 
n Low Levels of Rigor 



The most frequently earned grade for mathematics standards is a B+ (very rigorous), while the most 
frequently earned grade for the rigor of the English language arts standards is a C- (low levels of rigor). 
Figure 5 shows the distribution of CBE’s grades for the rigor of states’ mathematics and English language 
arts standards. 

Figure 5 

Grades for Rigor in States' Mathematics and English Language Arts Standards 
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It is not surprising that mathematics standards were found to be more rigorous than English language arts 
standards. Mathematics standards’ writers in the states had at their disposal the same models -- the NAEP 
mathematics framework and the national voluntary standards written by NCTM — on which CBE’s math- 
ematics framework was based. It is clear that many states used these models as guides, as they developed 
their own mathematics standards. 

Due to much controversy about the national voluntary standards in English language arts, writers of 
English language arts standards had no recognized model to use as a guide; consequently, it was expected 
that on the whole the scores for rigor in English language arts standards would not be as high as those for 
mathematics standards. This hypothesis was supported by our findings, which are presented in detail below. 



As mentioned earlier, to be rigorous, standards must address essential concepts and skills and must also 
require student understanding and application of these essential concepts and skills at a level of sophisti- 
cation or complexity that is appropriate and challenging to students at a particular grade level. We found 
that mathematics standards in most states include almost all essential concepts and skills, but vary in the 
level of sophistication required. 

The state English language arts standards documents, on the other hand, often simply lack many concepts 
and skills. Thus, our discussion of rigor in English language arts standards focuses on the missing essen- 
tial concepts and skills and possible reasons and ramifications. Our discussion of rigor in mathematics 
standards does this, but also analyzes particular framework benchmarks in detail. 

The Rigor in English Language Arts Standards 

As shown in Figure 6, of the 42 states’ English language arts standards, 7 were found to be very rigorous; 
21 were found to be rigorous; and 14 were found to contain low levels of rigor. 

Figure 6 



States with Very Rigorous English States with Rigorous English States with Low Levels of Rigor in 

Language Arts Standards Language Arts Standards English Language Arts Standards 



State 


Grade for Rigor 


State 


Grade for Rigor 


State 


Grade for Rigor 


Illinois 


A 


Massachusetts 


B 


Alabama 


C- 


Arizona 


B+ 


New York 


B 


Colorado 


C- 


California 


B+ 


West Virginia 


B 


Kentucky 


c- 


Michigan 


B+ 


Maine 


B- 


Minnesota 


c- 


New Hampshire 


B+ 


Missouri 


B- 


Mississippi 


c- 


Vermont 


B+ 


New Jersey 


B- 


Montana 


c- 


Wisconsin 


B+ 


Pennsylvania 


B- 


North Dakota 


c- 






Texas 


B- 


Oklahoma 


c- 






Virginia 


B- 


Tennessee 


c- 






Washington 


B- 


Utah 


c- 






Arkansas 


C+ 


Louisiana 


D+ 






Delaware 


C+ 


Indiana 


D 






Kansas 


C+ 


New Mexico 


D 






Rhode Island 


C+ 


Alaska 


D- 






Florida 


C 










Hawaii 


C 










Maryland 


C 










North Carolina 


C 










Ohio 


C 










Oregon 


C 










South Carolina 


C 
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Grades for the rigor of English language arts standards range from A to D-. The grades cluster around 
C-, with seven states receiving the grade of C, and ten states receiving the grade of C-. The average score 
for all states’ English language arts standards is 2.29, which translates into a grade of C+ (rigorous). 

Most state English language arts standards address basic skills. 

Most agreement among state standards documents occurred around the basic skills of English language 
arts. State standards documents, on average, scored high for fourth grade benchmarks that address reading 
using decoding and word attack skills and comprehension strategies. Reading in CBE’s framework and 
most state standards includes the expectation that by fourth grade students are adept and independent readers 
who recognize the many purposes for reading. Though there is currently debate over the benefits of phone- 
mic versus whole language reading strategies, this debate does not play a major role in our study. We have 
examined English language arts standards at fourth grade, by which time students should have already 
achieved literacy. However, CBE acknowledges the importance of providing all children with phonemic 
based reading strategies, as many studies have shown that this approach, in conjunction with others, works 
best for most students. 

There is also general agreement among state standards documents over the importance of requiring 
students to approach writing as a process, and to use correct grammar, punctuation, and spelling. Mauiy 
state English language arts standards include benchmarks about key writing processes — steps such as 
prewriting, peer review, revising, editing, and publishing — to develop and express ideas. It is also com- 
mon that state standards require students to employ proper language usage and mechanics. 

Many state English language arts standards do not address reading requirements, literature, research, 
and language study. 

States’ English language arts standards frequently omit, however, important concepts and skills in the areas 
of reading requirements, literature, research, and foundational language study. 

Specihc Reading Requirements 

In their analysis of state standards documents, English language arts evaluators detected frequent attempts 
to sidestep the controversial or politically sensitive. An unwillingness to specify what literature students 
are to read, to indicate the quality and quantity for reading expectations, or to advocate intellectualism 
translates into a lack of rigor in a state document. Some states do not mention anywhere within their documents 
whether, what, how much, and how frequently students are expected to read. These standards documents 
fail by not giving more particular guidance on an essential aspect of English language arts study. 

Two of CBE’s framework benchmarks that address reading state: 

Read and comprehend the equivalent of at least thirty books per year in first through fourth 
grade of a specified variety in their entirety, both assigned in class and chosen for independent 
reading (fourth grade). 

Read and comprehend the equivalent of at least thirty books per year in ninth through twelfth 
grade of a specified variety in their entirety, both assigned in class and chosen for independent 
reading (twelfth grade). 
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These “read a lot” benchmarks grew out of several firmly held beliefs that reading is the content of English 
language arts, the best readers are frequent readers, and that students should be encouraged at times to 
choose their own reading materials to promote reading as a lifelong skill. These benchmarks also emphasize 
the importance of providing a variety of reading experiences, to ward off the dulling practice of taking all 
classroom reading from a single source. 

Disappointing numbers of state standards documents require that students read a variety of texts; even 
fewer specify a quantity of reading. The average scores on the “read a lot” framework benchmarks for all 
42 documents (2.02 for fourth grade; 1.83 for twelfth grade) are among the lowest of all individual benchmark 
scores. It is possible that states left out this seemingly essential element so as to avoid having to enforce 
it. To make such a statement meaningful, the state, district, school, or teacher would need to document and 
assess each student’s progress to determine if the specified types and number of books have been read. 

Pointed statements about what students are to read are clear indicators of the level of rigor in a document. 
Reading Dostoyevsky requires a starkly different level of ability than reading Dr. Seuss. If a state standard 
requires students to “read competently,” it would be completely fitting for a first grader, but not for a 
twelfth grader, to read Dr. Seuss to demonstrate reading competency at an appropriate level of sophistication. 
Skirting the issue by using language such as, “Students read appropriate texts at appropriate levels,” may 
be accurate, but provides a very weak guide for teachers attempting to apply a state standards document 
or parents attempting to gauge their children’s progress. We cannot be sure that such a set of standards is 
rigorous. Though the English language arts framework included in Appendix C does not contain an author 
list, a suggested author list will be part of the English language arts document in CBE’s Standards for 
Excellence in Education (SEE), to be published in early 1998. 

Focused Literary Study 

At the twelfth grade, the framework benchmarks in literature are based upon a statement that focuses study 
of particular literature: 

Read a wide range of classic and contemporary literature from many places and periods and in 
many genres to build an understanding of the philosophical, ethical, and aesthetic dimensions of 
significant ideas and events. 

State standards tend to score very low on the framework benchmarks falling under this heading. The states 
that treat literature and reading with broad brushstrokes of statements miss giving particulars about the 
literature to be studied. For example, if a state shies away from demanding that students read substantially 
in any type of literature, then the state will likely not meet or match the rigor of our framework benchmarks. 
To conduct literary study of the depth suggested by our framework benchmarks, it is necessary that edu- 
cators make a philosophical commitment to the reading of works from particular time periods and cultures. 

Research 

Another area of study that is missing or weak in many state English language arts standards is research. 
CBE’s framework benchmarks include research skills such as giving precise credit for others’ ideas and 
gathering information from sources such as books, magazines, and CD-ROMs. Possible reasons for the 
exclusion of research-related standards are that states may not consider research an essential requirement 
for all students, or they may address research more thoroughly in the context of another content area, such 
as social studies. 
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Foundational Language Study 

An understanding of root words, word origins, and differences between standard English usage and slang 
are hallmarks of a thorough and rigorous English language arts education. Unfortunately, though, state 
average scores indicate that a foundational study of language is not highly valued by many states’ 
standards documents. Language study is not often a part of English curricula, so this choice by many states 
is not surprising. 

The Rigor in State Mathematics Standards 

Overall, the states’ mathematics standards were found to be quite rigorous. Whereas the grades for the 
rigor of the mathematics standards range from A to F, the states’ average score is a 2.88, which translates 
into a grade of B- (rigorous). Scores cluster around the B range, with 12 states earning the grade B+; 11 
states earning B; and seven states earning B-. 

As shown in Figure 7, 16 of the 43 states with mathematics standards were found to be very rigorous; 24 
were found to be rigorous; and three were found to contain low levels of rigor. 

Figure 7 



States with Low Levels 

States with Very Rigorous States with Rigorous of Rigor in Mathematics 

Mathematics Standards Mathematics Standards Standards 



State Grade for Rigor 


State Grade for Rigor 


State 


Grade for Rigor 


New Jersey 


A 


Kentucky 


B 


Nebraska 


C- 


New Hampshire 


A- 


Massachusetts 


B 


Alaska 


D- 


Utah 


A- 


Mississippi 


B 


Montana 


F 


West Virginia 


A- 


North Carolina 


B 






Arizona 


B+ 


Oklahoma 


B 






Colorado 


B+ 


Oregon 


B 






Connecticut 


B+ 


Rhode Island 


B 






Delaware 


B+ 


South Carolina 


B 






Kansas 


B+ 


Texas 


B 






Maryland 


B+ 


Virginia 


B 






Michigan 


B+ 


Washington 


B 






New York 


B+ 


Arkansas 


B- 






Ohio 


B+ 


Illinois 


B- 






Pennsylvania 


B+ 


Indiana 


B- 






Vermont 


B+ 


Maine 


B- 






Wisconsin 


B+ 


Minnesota 


B- 










Missouri 


B- 










New Mexico 


B- 










Florida 


C+ 










Georgia 


C+ 










Louisiana 


C+ 










Alabama 


C 










Hawaii 


C 










North Dakota 


C 
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States scored particularly well on several framework benchmarks representing concepts and skills that 
virtually all states consider vitally important. On average, state standards received a score higher than 3.60 
— at the upper end of the “very rigorous” scale — on the three framework benchmarks shown in Figure 8. 

Figure 8 

CBE's Mathematics Framework Benchmarks for which the States' Average Score is Greater than 3.60. 



Eighth Grade, Number Sense 

3. Add, subtract, multiply, and divide with rational numbers. (Average score for all states: 3.65) 

Eighth Grade, Measurement 

1. Select and use appropriate measurement instruments and units. 

(Average score for all states: 3.65) 

Eighth Grade, Geometry 

2. Given a figure, describe its geometric qualities. (Average score for all states: 3.65) 



Unlike English language arts standards, most states did not have major gaps in their mathematics 
standards. States with rigorous or very rigorous standards addressed almost all the essential concepts and 
skills in the framework benchmarks. These states’ standards tended to lose points when they were written 
at a slightly lower level of complexity than CBE’s framework benchmarks. However, only three states — 
New Hampshire, New Jersey, and Utah — addressed every benchmark in the mathematics framework and 
therefore received no scores of zero. Similarly, only five states — Michigan, Pennsylvania, Virginia, West 
Virginia, and Wisconsin — addressed all framework benchmarks but one. These states received just one 
score of zero. 

Four mathematics framework benchmarks were neglected by virtually all states. These benchmarks 
represent mathematical concepts and skills that are not valued highly or are often overlooked. Figure 9 
presents the framework benchmarks on which the states’ average score was lower than 2.00, reflecting an 
unacceptably low level of rigor. 

Figure 9 

CBE's Mathematics Framework Benchmarks for which the States' Average Score is Less than 2.00. 
Eighth Grade, Number Sense 

4. Describe the effects of operations (addition, subtraction, multiplication, division) on 
size and order of numbers. (Average score for all states: 1.88) 

Eighth Grade, Measurement 

5. Make and read scale drawings. (Average score for all states: 1.88) 

Twelfth Grade, Data Analysis 

4. Given a situation, describe a procedure for selecting an unbiased sample and identify 
the sources of sampling error. (Average score for all states: 1.79) 

Twelfth Grade, Communication 

3. Ask clarifying and extending questions related to mathematics. 

(Average score for all states: 1.98) 
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Taken as a whole, states’ mathematics standards did not have any notable strengths or weaknesses. Those 
interested in education reform have been debating whether to emphasize concepts (number sense, 
measurement, data analysis, geometry, and algebra) or skills (problem solving, communication, reasoning, 
and connections) in state standards and whether students should be required to use calculators. Fortunately, 
states seem to be taking the middle ground in the debate over concepts and skills; the difference in states’ 
average scores between concepts and skills was negligible. 

Based on a recommendation from our mathematics panel, CBE’s evaluation did not pass judgement on 
states’ inclusion of calculator use. CBE believes that all students should learn to compute without the use 
of a calculator, but calculators should be used to explore advanced mathematical concepts such as the 
graph of a polynomial function. CBE plans to incorporate the evaluation of the inclusion of calculators in 
state standards in future studies. 

Those state standards that contain low levels of rigor tend to be weak in all areas of their mathematics 
standards, but two of the three states’ standards were especially weak in the eighth-grade framework 
benchmarks. One state received a score of “0” on every eighth-grade framework benchmark (as compared 
to its average of 1.4 for the twelfth-grade framework benchmarks), and another received an average score 
of 0.37 for all eighth-grade framework benchmarks (as compared to an average score of 0.7 for the twelfth- 
grade framework benchmark). Across all states, the results indicate that eighth-grade standards (total 
average grade of B) were slightly more rigorous than twelfth-grade standards (total average grade of B-). 

Using the Findings to Improve State Standards 

The results of this study show states exactly which parts of their standards documents are rigorous and 
which parts need improvement.^ Because CBE is able to offer such precise information, states will be able 
to act quickly to strengthen their standards. For some states such improvements might be as straightfor- 
ward as requiring students to conduct more research as part of their English language arts training; for 
others, they might be as complicated as rewriting standards for an entire subject. Conversely, states might 
be able to explain why they believe that improvements to their standards are not needed, despite a low 
grade from CBE’s analysis. For example, a state may point out that it includes research expectations in its 
social studies standards and therefore does not consider it necessary to print them again in English 
language arts standards. Whatever the consequences for each state, it is our hope that the study will 
foster a greater understanding of rigor in standards. 



‘Individual state data have been sent to chief state school officers and presidents of the school boards in each state. 
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III. Issues Affecting the Evaluation of Rigor in State Standards 



In developing and refining our process for analyzing the rigor in state standards, CBE staff and panel 
members confronted several critical issues. This section details CBE’s decisions regarding these issues. 

Specificity is not equal to, but is an aspect of rigor. 

As we struggled with the definition of rigor, we found it necessary to distinguish between specificity and 
rigor. For the purposes of our analysis, CBE has defined specificity as language that describes what is most 
essential for students to learn, using sufficient detail to convey what is expected without dictating instruc- 
tional strategies. Specificity does not equal rigor; state standards might be very specific but omit major 
concepts and skills or require mastery at an insufficiently low level of sophistication. 

CBE’s analysis of rigor incorporated specificity. CBE’s framework benchmarks are written in language 
that is clear and specific about what students should know and be able to do at certain points in their 
educational careers. To be deemed rigorous, state standards must contain similar specificity in their standards 
in order to compare favorably with the framework benchmarks. One level of CBE’s scoring rubric is 
designed to address overly vague and nonspecific language in state standards (see rubrics in Appendix D). 

CBE^s analysis examined standards for two grade levels. 

In our analysis, standards at the eighth and twelfth grades were analyzed for mathematics and at the fourth 
and twelfth grades for English language arts. We evaluated the standards from the highest grade, or exit 
standards, because these standards define a state’s expectation for student learning until graduation. We 
chose to assess standards of an earlier grade to determine how a state’s expectations progress. 

The most equivalent grade level standards were used in our analysis. 

In addition to language specificity, grade level specificity, which refers to the number of grades for which 
standards are written in a state document, is an important consideration for standards’ writers. For example. 
New York has written standards for the elementary, intermediate, and commencement levels, whereas 
Virginia has written standards for every grade, kindergarten through grade 12. 

In evaluating the standards for rigor, we took the standards written for the examined grade level and all 
prior grade levels into account. For example, in evaluating a state’s eighth-grade mathematics standards, 
CBE examined all of the state’s standards written for kindergarten through grade 8. 

Not all state standards documents have standards at the grade levels we examined. If a state’s standards 
are written for grades other than the ones examined by CBE, we used the closest equivalent standards, 
always giving the state the benefit of the doubt. For example, one state’s English language arts standards 
are divided into clusters of End-of-Grade 3, End-of-Grade 6, and End-of-Grade 10. To evaluate the rigor 
of this state’s fourth-grade standards, CBE examined the state standards written through the End-of-Grade 
6 cluster. We avoided making arbitrary decisions about what was or was not included in the fourth, rather 
than the fifth or sixth grades. As a result, the state’s English language arts standards probably scored higher 
than is truly accurate for its fourth grade level of rigor. 

There are pros and cons for writing standards at every grade level or in grade clusters. The advantage of 
including standards for every grade level is that everyone — administrators, teachers, students, parents, and 
other interested citizens — knows what is expected of students during each school year. Setting standards 
for every grade level simplifies the process of aligning curriculum to standards. It also eliminates confusion 
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among teachers about what they can expect students to have learned the year before and what they must 
prepare their students to be ready to learn the following year. Parents will know exactly what material their 
children are learning and can participate readily in their children’s education. Students also will be privy 
to a clear set of expectations for each academic year. 

The disadvantage of establishing standards for every grade is twofold. The first is that research shows that, 
until children are in about grade three, what is considered normal cognitive development encompasses 
a fairly wide range. This normal and elastic course of child development is incompatible with highly 
specific standards for grades K, 1,2, and 3. If a child is not ready to meet some first grade standards until 
the second or third grade, it is probably inappropriate to say that the child has failed. Doing so might give 
the child an early and needless sense of failure, which might prove difficult to overcome in later years. 
Further, teachers might be prompted to place such pupils into less challenging tracks, further delaying their 
academic growth. 

A second disadvantage of grade-by-grade standards becomes evident in a student’s later years in school. 
In high school, students traditionally enjoy some flexibility, choosing in which year they will study certain 
material. One science course or another may be taken in the senior year; a particular history course may 
be taken in tenth or eleventh grade. Under such a system, pegging standards to a particular year, rather than 
to a particular course, would not make sense. 

If a state does not have standards for grade 12, CBE evaluated its exit standards, or the knowledge and 
skills the state requires by graduation. Often states write exit standards for tenth grade. This approach gives 
college-bound students two years of college-preparatory classes, and students experiencing difficulties two 
more years to meet the standards. 

CBE evaluated only those standards that define expectations for all students and not those written for 
special tracks or groups of students. Therefore, CBE did not evaluate any optional standards or standards 
written especially for college-intending students. 

CBE searched for equivalent rigor, rather than equivalent language. 

In conducting our analysis of state standards, we were not looking merely for language that matched 
CBE’s framework benchmarks. States were not penalized for using wording different from that of CBE’s 
framework benchmarks or for using more, disparate, or fewer statements to convey standards. What was 
of utmost concern to evaluators was determining whether the state standards included the essential 
concepts and skills at a sufficiently high level of sophistication. 

Technology was not a part of our analysis. 

Technology is an important tool for student learning. State education departments are devoting large sums 
of money to wire schools for computer use, train teachers to use current technology, and introduce tech- 
nology into curricula. The majority of states incorporate references to technology in their standards. Some 
do so within the content-specific standards, whereas others dedicate a separate section to technology standards. 

CBE did not examine technology standards in the analysis, because there was disagreement within our 
advisory panels about how technology should be assessed. In addition, although CBE views technology as 
an important tool for learning, we have not come to conclusions about how best to include technology in 
content-specific standards. We plan to make a decision about whether and how to include an analysis of 
technology-related standards for next year’s study. 
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Only standards^ and not supporting material, were examined in CBE^s analysis. 

Since every state uses varied terminology, formats, and levels of detail, it was important that we be 
consistent and fair in defining which statements were to be used in the analysis. 

Statements selected for comparison and evaluation from each state were similar in their levels of abstraction 
and in content. CBE used, as points of comparison, the most specific statement from each state’s document 
that describes what students should know or be able to do. 

States sometimes include in their standards documents examples of student work or suggested instructional 
activities to aid the readers’ understanding of standards and how they can be applied. While helpful to 
teachers and students, this material was not used in our evaluation, because it is not clear that all 
students will be required to produce such work. 

Standards must address both concepts and skills. 

Standards should contain an appropriate balance of concepts and skills. It is essential, for example, that 
mathematics standards include major concepts, such as geometry, algebra, and data analysis (statistics and 
probability). They should also include skills, such as problem solving, communication, and reasoning, 
which allow students to apply those concepts. Although the distinction is less clear in English language 
arts, concepts include aspects of literature, such as plot, character development, and theme, as well as 
grammar and language usage rules, whereas skills include editing*, researching, and speaking. 

The current focus on “learning to think” and “developing critical thinking skills” has led some standards’ 
writers to create documents that are weak in the content knowledge required. Others, correctly noting that 
skills matter little if one has no understanding of core concepts to which to apply them, go to the other 
extreme, overloading standards documents with concepts but neglecting to ensure that students do more 
than memorize facts. CBE staff and advisory panels believe that a rigorous set of standards should include 
both the essential concepts and related skills for each subject. This belief is captured in CBE’s mathematics 
and English language arts frameworks. 

Steps were taken to ensure accuracy, validity, and reliability. 

As illustrated in Figure 3 in Section II, states are in various stages in the development of standards; a pri- 
ority of CBE’s analysis was to evaluate only state standards that are currently in use or will be adopted as 
of January 1998. State superintendents and school board presidents were contacted by CBE staff and also 
by the staff of Education Week to assure that the documents scored were the most up-to-date and official 
standards documents in use. 

The evaluation process was designed to be analytic, quantitative, and replicable. These goals guided CBE 
staff and the advisory panels in the development of the frameworks and scoring rubrics. To assure 
reliability among evaluators, all participated in numerous training and calibration sessions and met 
frequently to discuss complicated scoring issues. 
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IV. Application of the Evaluation Process 



After three full days of training (see Section V), each evaluator independently scored approximately ten 
state standards documents. To do so, evaluators first looked for standards from each state document being 
scored that were comparable to a given framework benchmark (8 1 total benchmarks in mathematics and 
62 total benchmarks in English language arts). If there was no standard in the state document that 
compared to the framework benchmark, the state document received the score of “0” for that benchmark. 

If there were state standards comparable to the framework benchmark, the state standards received a score 
of one, two, three, or four for that benchmark. To determine which score the state standards received, the 
evaluator used the scoring rubric for either mathematics or English language arts (see Appendix D for 
scoring rubrics). To receive a four, the standard(s) had to include all of the ideas expressed in the frame- 
work benchmark at a comparable level of sophistication. A state’s standards received a lower score if they 
did not include all ideas or else included all ideas but at a lower level of sophistication. 

To arrive at a final grade for the rigor of a state standards document, the individual scores of each 
individual framework benchmark were averaged. This averaged score (for example, 3.24) was converted 
into a letter grade, as is shown in Figure 10. 



Figure 10 



Assignment of Grades to Final Averaged Scores 


4.0> A >3.7 


3.7> A- >3.5 


3.5> B+ >3.2 


3.2> B >2.9 


2.9> B- >2.5 


2.5> C+ >2.2 


2.2>C>1.9 


1.9> C- >1.5 


1.5> D+ >1.2 


1.2> D>0.9 


0.9> D- >0.7 


0.7> F 



With a clear, analytical, and quantitatively-based evaluation process in place, scoring the state standards 
was relatively straightforward. This section presents examples of state standards for both mathematics and 
English language arts that received each of the possible scores on the rubric, as well as brief explanations 
for why the scores were given. The goal is to illuminate the scoring process, not to rationalize every score. 

Under each possible score (one, two, three, and four) an example first from mathematics and then English 
language arts is given.^ For each subject, a benchmark from the scoring framework is presented. Then, 
choosing a state at random (all states, regardless of their final grade, received at least one of each score), 
the sample state standards that align with the framework benchmark are listed, followed by a brief expla- 
nation for why the score was given. Readers may consult the rubrics in Appendix D to validate the scores 
and understand the process. 

Examples of Standards that Received the Score of One 

Scores of one and zero (given only when a state’s standards document contains no standards that align with 
a particular framework benchmark) are the only scores that represent an unacceptable level of rigor in the 
state standards. 



^Examples for the score of zero are not given because by definition, framework benchmarks earning a zero have no parallel state standards. 
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Figure 11 



Mathematics Example: Score of One 

Framework benchmark (twelfth grade, data State standard(s) aligning with benchmark: 

analysis, #4): Given a situation, describe a Design a statistical model to study a problem, 

procedure for selecting an unbiased sample 
and identify the sources of sampling error. 

^ Explanation of score: This state standard is the only one in the document that might be said to 
pertain to selecting an unbiased sample and identifying the sources of sampling error. While 
it could be argued that designing a statistical model to study a problem requires an under- 
standing of sampling, students could easily meet the state's standard without such an under- 
standing, let alone the ability to apply that understanding to a given situation. For this rea- 
son, the evaluator decided that while the state standard loosely addressed the ideas of unbi- 
ased sampling and sampling error, the standard omitted the main concepts and required stu- 
dent learning at a very low level of sophistication. 



Figure 12 

English Language Arts Example: Score of One 



Framework benchmark (fourth grade. State standard(s) aligning with benchmark: 

strand I, #1): Read aloud independently with Exhibit fluency in reading, 
accuracy, fluency, and comprehension, using 
decoding and word attack skills (e.g., phono- 
logical awareness, phonic analysis, structur- 
al analysis) and comprehension strategies 
(e.g., inferencing; predicting, summarizing, 
and paraphrasing or retelling; generating 
questions; and identifying main ideas and 
details to support them). 

Explanation of score: The state standard could generally be said to address one of the essential 
concepts or skills in the framework benchmark but at an unacceptably low level of sophistica- 
tion. The expectation that students read fluently is an important element of CBE's benchmark, 
but there are critical elements missing from the state standards: that students are to read 
aloud, that they are to read accurately, and that they are to comprehend what they read. In 
addition, the state's standards do not address techniques students may employ to become 
fluent readers. 
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Examples of Standards that Received the Score of Two 

A score of two is the lowest score given to state standards deemed to contain an acceptable level of rigor. 
State standards scoring a two, however, are still missing essential concepts or skills, require the 
understanding of concepts or application of skills at a significantly lower level of sophistication than the 
framework benchmark, or some combination of the two. A score of two was also given to state standards 
that seemed to have the essential concepts or skills and an appropriate level of complexity, but required 
the evaluator to interpret meaning and assume intent to a large degree. In other words, the level of 
specificity in state standards can affect rigor if it remains unclear whether a given concept or skill is 
required by the standard. 



Figure 13 

Mathematics Example: Score of Two 



Framework benchmark (twelfth grade, algebra, 
#7): Use right triangle trigonometric func- 
tions to model real-world phenomena. 


State standard(s) aligning with benchmark: 

Use trigonometric ratios in problem-solving 
situations (for example, finding the height of 
a building from a given point, if the distance 
to the building and the angle of elevation 
are known). 


Explanation of score: The state's expectation that students use trigonometric ratios clearly 
addresses the framework benchmark's right triangle trigonometric functions, but at a signifi- 
i cantly lower level of sophistication. The concept of functions (e.g., f(x)=3 sine (x)) is potentially 
; far more difficult than the concept of ratio. In addition, the framework benchmark requires stu- 
dents to use functions to model phenomena, whereas the state standard requires students 
I only to apply a formula to solve a well-defined situation (lower level of sophistication). 


Figure 14 




English Language Arts Example: Score of Two 




Framework benchmark (twelfth grade, strand IV, 
#2): Modify message, delivery, 
and organization according to the verbal 
and nonverbal signals perceived during 
oral presentations and group discussion. 


State standard(s) aligning with benchmark: 
Students are required to monitor and adjust ; 
their own oral and written presentations ! 

to have the greatest influence on a 
particular audience. 


Explanation of score: This standard earned a score of two because the evaluator had to make 
too many assumptions about the meaning of the standard to determine alignment. It is not 
clear what about their presentations or audience reaction students are expected to monitor in 
order to give effective presentations. The framework benchmark is more specific in delineating 
that students need to be prepared to modify message, delivery, and organization and that 
verbal and nonverbal audience cues may inform such a modification. 
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Examples of Standards that Received the Score of Three 

A score of three was given to state standards that align closely with a framework benchmark, but omit an 
essential concept or skill (or a few minor components), require learning at a slightly lower level of sophis- 
tication, or some combination of the two. 

Figure 15 



Mathematics Example: Score of Three 



1 Framework benchmark (eighth grade, measure- 
ment, #5): Make and read scale drawings. 


State standard(s) aligning with benchmark: 
Read and interpret various scales including 
those based on number lines and maps. 


Explanation of score: Both the framework benchmark and the state's standards require 
students to understand scale drawings, representations of real-world objects or places using 
drawn lines and spaces consistently to denote larger areas (for example, a map might 
represent an one mile road with an one-inch line). Whereas the framework benchmark 
and the state standard both require students to read scales, the framework benchmark also 
! expects students to make them; therefore, the state standard, while quite rigorous is 
missing an essential concept. 


Figure 16 




English Language Arts Example: Score of Three 




Framework benchmark (twelfth grade, strand 
VII, #1): Analyze an author's point of view, 
purpose, and historical and cultural context 
in relation to the text's meaning. 


State standard(s) aligning with benchmark: 

• Analyze how the works of a given period 
reflect historical events. 

• Acknowledge a diversity of ideas generated 
by authors of different race, belief, gender, 
age, etc. 

• Analyze literature for the purpose, ideas, 
and style of the author. 

• Read from and understand different points 
of view. 


; Explanation of score: These standards are a quite close match to the framework benchmark, 

! yet some elements occur at a lower level of sophistication. Students are not asked to analyze 
the cultural influence on an author's choices or to analyze an author's point of view. In addition, 
there is no specified mention of relating this analysis to a text's meaning. 
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Examples of Standards that Received the Score of Four 

A four was given to state standards that align perfectly to a framework benchmark. 



Figure 17 

Mathematics Example: Score of Four 



Framework benchmark (eighth grade, 
geometry, #3): Predict results of combining, 
subdividing, and changing shapes. 


State standard(s) aligning with benchmark: 
Analyze and create new shapes by 
combining, dissecting, or transforming 
existing shapes. 


Explanation of score: This state's standard requires all of the concepts and skills in the frame- 
work benchmark at a comparable level of sophistication. 


Fiaure 18 




English Language Arts Example: Score of Four 




Framework benchmark (fourth grade, 
strand IV, #7): Generate and display ideas, 
information, and images. 


State standard(s) aligning with benchmark: 

• Locate and compile information from 
several sources and arrange it logically to 
communicate ideas clearly. 

• Demonstrate a working knowledge of 
technology by producing multimedia pre- 
sentations and by accessing resources 
through the school's network. 

• Develop presentations in creative formats 
such as videotapes and slides; create 
original works for sharing and publishing. 


Explanation of score: The state standards require mastery of all of the elements mentioned in 
the framework benchmark at a comparable level of sophistication. Students are asked to gen- 
erate presentations using a variety of technological resources. The presentations are to 
include gathered and organized information. "Original" works implies that students are to 
generate their own ideas for presentation. Images may appear in various forms such as in 
videotape or slide demonstrations. 
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V. Process Design 



CBE created a process for defining rigor and applying that definition to evaluate standards that is fair, 
reliable, analytical, and explicit, and therefore replicable and informative. The evaluation process 
consisted of eight steps: 

1. Development of panels 

Because public acceptance of the definitions of rigor and the process used to evaluate state standards is 
valued by CBE, we created two advisory panels — one for mathematics and one for English language arts. 
Each included subject area experts, a parent representative, teacher representatives, and a business 
community representative (see Appendix A for a list of panel members). These panels were responsible for 
overseeing CBE’s work and making recommendations for improvement. 

2. Development of draft definitions, frameworks, and rubrics 

Panelists were asked to review CBE-drafted definitions of rigor, frameworks, and rubrics. Both advisory 
panels made minor changes to the definitions of rigor and offered valuable suggestions for improving the 
frameworks, scoring rubrics, and scoring process. 

Frameworks 

CBE’s definition for rigor references the National Assessment of Educational Progress (NAEP) and the 
National Council of Teachers of Mathematics (NCTM) frameworks in mathematics and the Standards for 
Excellence in Education in English language arts. These frameworks were adapted to meet the needs of 
our analysis. 

Mathematics 

NAEP is a congressionally mandated project of the U.S. Department of Education’s National Center 
for Education Statistics. The tests assess how well U.S. students meet the NAEP-defmed standards in 
mathematics and other academic subjects. Since 1969, NAEP has surveyed the achievement of students at 
ages 9, 13, and 17, and since the 1980s, in grades 4, 8, and 12. 

The purpose of the NAEP mathematics framework is to serve as the basis for a national assessment of 
student learning in mathematics in the United States. This assessment is the only one of its kind, and is 
authorized by the U.S. Congress and funded by the federal government. The assessment provides indica- 
tions of student achievement in mathematics at each grade level assessed and provides comparisons among 
student populations with various background characteristics (e.g., race/ethnicity, gender, parents’ educa- 
tional level). NAEP results also may be used to determine trends in student achievement and to show state- 
by-state comparisons. The NAEP mathematics framework identifies what students should know and be 
able to do in math, and is designed to serve as the basis for developing the actual assessment or test items 
to which students respond. 

The NAEP framework includes the concepts (number and computation, measurement, statistics and prob- 
ability, algebra, and geometry) of the voluntary national mathematics standards, developed by the National 
Council of Teachers of Mathematics (NCTM) in 1989.® The NAEP framework, however, does not 
delineate or provide much detail about mathematical skills (e.g., problem-solving, reasoning, mathematical 
communication, and mathematical cormections), deemed essential by NCTM. For this reason, CBE and 
the mathematics advisory panel decided to add NCTM’s skills standards to NAEP’s concept strands to 
create CBE’s framework benchmarks. 



^National Council of Teachers of Mathematics, Curriculum and Evaluation Standards for School Mathematics, Reston, VA: NCTM, 1989. 




O -A 



O U 



Process Design/27 



Once CBE combined all benchmarks from NAEP and NCTM, several changes needed to be made to use 
the amalgam as a framework for evaluating the rigor of states’ mathematics standards. CBE wanted to 
ensure that all framework benchmarks were appropriate for all students. The NAEP framework purpose- 
fully includes items that are too difficult for many students, because it is used to develop an assessment 
intended to sort students by achievement. Whereas sorting might be important for the purposes of NAEP, 
it runs counter to the intent of academic standards, which are meant to define what all students should 
know and be able to do. For this reason, CBE and the mathematics advisory panel made several changes 
to the NAEP framework to develop CBE’s mathematics framework. For example, CBE took out NAEP’s 
reference to complex numbers, citing this concept as overly rigorous to require for all students. 

CBE also wanted to ensure that each benchmark met all of CBE’s criteria for high-quality standards. 
Although our focus was rigor, CBE discarded potential framework benchmarks if they did not meet CBE’s 
other criteria for high-quality standards. For example, CBE staff omitted an NCTM skills benchmark that 
reads, “Acquire confidence in using mathematics meaningfully.” This benchmark does not meet CBE’s 
criteria for high-quality standards, because it is difficult to measure and is based on values rather than 
mathematical concepts or skills. 

Some of the benchmarks adapted from NAEP and NCTM were much more rigorous than others. For exam- 
ple, one of NAEP’s benchmarks incorporated in CBE’s framework is: “Solve equations and inequalities 
algebraically, graphically, and using matrices.” This benchmark carries a great deal of weight, representing 
a significant proportion of what students should learn in an entire semester of algebra. In contrast, one of 
NCTM’s benchmarks is: “Formulate counterexamples.” This benchmark does not compare in importance 
or breadth to the example from NAEP. For this reason, CBE decided to combine this benchmark with 
a similar benchmark to create a framework benchmark that reads, “Judge the validity of arguments, 
formulating counterexamples as necessary.” 

English Language Arts 

Unlike the NAEP framework for mathematics, NAEP frameworks for reading and writing cannot stand 
alone as a standards document, although almost all learning goals in the NAEP frameworks appear in 
CBE’s framework benchmarks. Whereas many essential concepts and skills are addressed in the NAEP 
frameworks, they are often described in overly general terms. Some areas of study essential to English lan- 
guage arts learning, such as listening, speaking, and research, are not addressed in the current NAEP 
frameworks. For example, the writing assessed by NAEP is first draft writing. A critical element of 
learning to write involves spending time developing a topic, outlining plans for writing, revising drafts, 
and receiving and providing feedback on writing. Such detailed steps are not incorporated into the 
present NAEP frameworks. 

In 1991, two professional organizations, the National Council of Teachers of English (NCTE) and the 
International Reading Association (IRA) launched an effort to write model voluntary national English 
standards. The NCTE/IRA standards for English language arts, however, have been beset with problems 
and controversy. The Department of Education stopped funding the NCTE/IRA effort after the groups had 
spent close to $2 million over three years and NCTE/IRA had failed to reach agreement or produce much 
workable documentation. NCTE/IRA was then given private funds to complete the work, but their final 
product has been criticized since its introduction for being overly broad and therefore unusable. 

CBE and the English advisory panel chose to use the Standards for Excellence in Education (SEE) as the 
basis for the English language arts framework. SEE is a CBE project, initiated in July 1995, to create model 
standards for each content area. The SEE English language arts document was written in consultation with 
subject area experts who drew inspiration from exemplary state and national standards documents. 
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Scoring Rubric 

The mathematics and English language arts frameworks were designed as the models against which state 
standards would be compared. State standards rarely compared perfectly with the framework benchmarks, 
however. Frequently, state standards contain some of the ideas contained in the framework benchmarks or 
require the understanding of content or the application of skills at a level of sophistication that is 
comparable, but not as high as, the framework benchmarks. To guide our evaluation of state standards, 
CBE created rubrics, scoring guides that list criteria for evaluating the state standards and assigning points 
depending on the degree to which the criteria are met (see Appendix D). 

The rubrics enable evaluators to score state standards consistently, using defined reasons for assigning par- 
ticular scores. CBE’s evaluation team consisted of four evaluators for mathematics and five for English 
language arts, each of whom is a current or former teacher in the content area evaluated. Rubrics and train- 
ing increased the likelihood that all evaluators scored the state standards with the same criteria and deci- 
sion rules and arrived at scores that are valid, reliable, and fair. 

3. Presentation of drafts to panels 

CBE presented expert panels with draft definitions, frameworks, and rubrics. Panel members provided 
advice about how to improve the drafts and evaluation process. Figure 19 summarizes the major recom- 
mendations from the panels and the subsequent actions taken by CBE: 



Figure 19 

Panel Recommendations and Subsequent Action Taken by CBE 



Mathematics 




English Language Arts 


Panel 

Recommendation 


CBE Action 


Panel 

Recommendation 


CBE Action 


Revise the definition 
of rigor. 


Definition was revised 
by CBE and approved 
by panel. 


Revise the English 
framework. 


Framework was revised 
by CBE and approved 
by panel. 


Require evaluators to 
have sound 
mathematical 
knowledge base. 


All CBE evaluators 
are current or former 
mathematics teachers, 
well versed in 
the subject. 


Include in the rubric 
criteria for whether a 
concept or skill is 
listed explicitly by 
the state or merely 
implied. 


Rubric was revised 
by CBE and approved 
by panel. 


Facilitate a discussion 
among the evaluators 
prior to analysis to 
agree howto interpret 
various meanings of 
words in state standards 
and framework 
benchmarks. 


CBE conducted 
extensive training 
of evaluators. 


Clarify the difference 
between specificity 
and rigor and 
examine how the 
characteristics relate. 


Rubric was revised to 
explain how specificity 
might influence a 
definition of rigor. 


Do not include 
technology in the 
framework; consider 
incorporating it in 
future analysis. 


Did not include 
technology in 
framework. 
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4. Evaluators’ preparation for the scoring process 

CBE staff worked together to develop agreement about how the definitions, frameworks, and rubrics 
would be applied to evaluate the rigor of state standards in a way that was fair, consistent, reliable, and 
meaningful. During several days of extensive training, evaluators reached consensus on how to interpret 
the definitions, frameworks, rubrics, and state standards. Mathematics evaluators and English language 
arts evaluators participated in similar training, although they did so separately. Two of the evaluators (one 
representing each subject team) were responsible for ensuring consistency and cohesion across subject- 
area training. Evaluators used a six-step process to calibrate the scores: 

1. Group scoring. All evaluators scored one-half of a state standards document together, arriving at 
consensus on scores and discussing confusing issues. 

2. Individual scoring. Next, each evaluator independently scored the same two state standards documents. 

3. Group consensus. The evaluators then convened and sought consensus on each set of scores for 
both state documents. Along the way, evaluators learned whether they were misinterpreting 
specific items, scoring certain cases incorrectly, or generally scoring too high or too low. 

4. Individual scoring. Each evaluator then independently scored two more states’ standards documents. 

5. Partner confirmation. Evaluators met in pairs to reach consensus on the scores for a state and to 
discuss contentious scoring issues. Evaluators then switched partners to reach consensus on the 
scores for the second state document that each evaluator scored independently and to discuss 
other issues. 

6. Group discussion. Finally, all of the subject evaluators met together to finalize decisions about scoring. 

5. Panel review 

After the training, each subject team sent its panel members the rubric, framework, a sample state’s 
standards document for the relevant subject, and a scored evaluation of that state’s standards. Panelists 
raised several concerns and offered specific recommendations for how to improve the scoring process. 

6. Incorporation of panels’ feedback 

The evaluators synthesized the panel members’ feedback and made changes to the frameworks and 
scoring process when appropriate. 

7. Final calibration by evaluators 

After changes were made, evaluators met again to ensure that they had understood the changes and how 
those changes would affect scoring. State standards that had been scored in the earlier training were scored 
again, using the revised framework and scoring process. 

8. Scoring of state standards 

Evaluators scored the remaining and the bulk of state standards documents independently, but conferred 
in situations where scoring questions arose. 
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VI. Future Directions 



Remaining Questions 

In the course of the analysis, we learned a great deal about the rigor of state standards; however, key issues 
remain that, once addressed, will enrich the discussion of the value and quality of standards across the 
nation. To provide states with a more comprehensive look at their standards’ strengths and weaknesses, 
CBE seeks to synthesize the findings about the rigor of standards with answers to related and pertinent 
questions regarding the overall quality of standards: 

• Are they clearly written? 

• Are they well organized? 

• Are they measurable? 

• Are they accepted within the state, districts, and schools? 

• Are the standards being implemented effectively? 

• Are the standards free of bias? 

• Are the standards developmentally appropriate at each grade level? 

Future Plans for Analyzing Rigor in Standards for All Subjects 

CBE believes that all students should receive a strong academic grounding in mathematics, English 
language arts, civics, geography, history, science, the arts, and a foreign language. In our first year of 
participation in Education Week's Quality Counts '98 project, CBE took on the manageable goal of assess- 
ing rigor of state standards in only two of these subject areas, mathematics and English language arts, at 
two grade levels. An analysis of standards in these subjects does not give a complete picture of learning 
expectations within a state. The rigor of the standards from different subjects varies widely, even for a par- 
ticular state; consequently, the evaluation of the rigor of a state’s mathematics and English language arts 
standards should not be extrapolated to other subjects. Next year, CBE plans to expand the scope of the 
analysis to include science and history/social studies. 

Standards and Student Achievement 

A rigorous set of standards does not translate directly into high student achievement. A state’s standards 
set expectations for student learning but do not ensure that these standards are met. Once states have had an 
opportunity to implement standards, it is hoped that student achievement will begin to match the expectations. 

Standards’ Role in Education Improvement 

Evaluation of state standards documents, no matter how comprehensive that evaluation may be, can only 
begin to measure the overall effectiveness of standards. What ultimately determines the quality of a set of 
standards is whether the students in the classrooms are learning, whether the teachers are equipped to 
address the learning needs of their students, and whether the community is supportive of these education 
goals. What the best standards can do is provide clear indications of what students should know and be 
able to do at various points in their educational careers, what teachers should expect of students, and what 
parents should expect of their children and the schools they attend. 

If standards-driven reform is to live up to its own great expectations, it is critical that states’ standards be 
high but not unreachable, and specific but not directive. Standards should be clear to all readers, whether 
they be educators, parents, or students. Teachers must demand that their students meet the standards and 
must provide the instruction and support students need. 
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But adherence to standards should not rest on teachers’ shoulders alone. Parents must ensure that their chil- 
dren are ready to learn, provide home environments conducive to learning, and support their children as 
they work to meet the standards. Administrators must support teachers with training and resources. 
Districts and states need to administer assessments to determine whether students are meeting the stan- 
dards, and to implement policies for students and schools that fail to do so. Finally, no set of standards is 
ever complete. A review process should be built into the standards system to allow for continuous improve- 
ment of all its components. 
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Appendix A 



Advisory Panels 



Panelists for the Mathematics Advisory Council 

Bernard Cotton 

President, Glasgow PTA 

Alice Gill 

Associate Director, Educational Issues Department, American Federation of Teachers 

Milt Goldberg 

Executive Vice President, National Alliance of Business 

Cindy Hannon 

Mathematics Specialist, Maryland State Department of Education 

Craig Jerald 

Project Director, Quality Counts ‘98 

J. Arthur Jones 

Senior Associate, Quality Education for Minorities 

Sam Rankin 

Director, American Mathematical Society 
Andy Reeves 

Director of Editorial Services, National Council of Teachers of Mathematics 

Douglas Tyson 

Math and Science Teacher, Benjamin Banneker High School 
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Panelists for the English Language Arts Advisory Council 



Cleve Bryant 

Humanities and English Teacher, Georgetown Day School 

Ruth Dennis 

Assistant Director, Educational Issues Department, American Federation of Teachers 

Alan Farstrup 

Executive Director, International Reading Association 

Bonnie Hain 

English Language Arts Specialist, Maryland State Department of Education 

Craig Jerald 

Project Director, Quality Counts ‘98 

Gene Kijowski 

President, Century Pools 

Louisa Moats 

Project Director, Washington, D.C. Early Interventions Project, National Institute of Child Health and 
Human Development 

Shirley Nelson 

President, Fairfax County Council of PTAs 
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Appendix B 



Grades for the Rigor of State Standards 



State 


Math 


ELA 


Alabama 


C 


C- 


Alaska 


D- 


D- 


Arizona 


B+ 


B+ 


Arkansas 


B- 


C+ 


California 


** 


B+(*) 


Colorado 


B+ 


C- 


Connecticut 


B+(*) 


** 


Delaware 


B+ 


c+ 


Florida 


C+ 


c 


Georgia 


C+(*) 


** 


Hawaii 


c 


c 


Idaho 


** 


** 


Illinois 


B- 


A 


Indiana 


B- 


D 


Iowa 


*** 


*** 


Kansas 


B+ 


C+ 


Kentucky 


B 


C- 


Louisiana 


C+ 


D+ 


Maine 


B- 


B- 


Maryland 


B+ 


C 


Massachusetts 


B 


B 


Michigan 


B+ 


B+ 


Minnesota 


B- 


C- 


Mississippi 


B 


C- 


Missouri 


B- 


B- 



* Reviewed a draft of the standards that is expected to be 
approved by the State Board by January 1998. 

** Standards under development 
*** No state standards. 



State 


Math 


ELA 


Montana 


F 


C- 


Nebraska 


C- 


** 


Nevada 


** 


** 


New Hampshire 


A- 


B+ 


New Jersey 


A 


B- 


New Mexico 


B- 


D 


New York 


B+ 


B 


North Carolina 


B 


C 


North Dakota 


C 


C- 


Ohio 


B+ 


C 


Oklahoma 


B 


C- 


Oregon 


B 


C 


Pennsylvania 


B+(*) 


B-(*) 


Rhode Island 


B 


C+ 


South Carolina 


B 


C 


South Dakota 


** 


** 


Tennessee 


** 


c- 


Texas 


B 


B- 


Utah 


A- 


C- 


Vermont 


B+ 


B+ 


Virginia 


B 


B- 


Washington 


B 


B- 


West Virginia 


A- 


B 


Wisconsin 


B+(*) 


B+(*) 


Wyoming 


*** 


*** 
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Appendix C 

Frameworks for Mathematics and English Language Arts 

Mathematics Framework 

I Eighth Grade 

I. Number Sense, Properties, and Operations Score 

1. Generate, compare, and use equivalent representations of rational numbers 

(including scientific notation and exponents) for given situations. 

2. Read, write, rename, order, model, and compare rational numbers. 

3. Add, subtract, multiply, and divide with rational numbers. 

4. Describe the effects of operations on size and order of numbers. 

5. Describe features and steps associated with algorithms. 

6. Select an appropriate computation method (pencil and paper, calculator, 

mental arithmetic). 

7. Know when and how to round and estimate in meaningful contexts with 

whole numbers, decimals, and fractions. 

8. Use ratios and proportions to model and solve problems 

(including rates, scaling, and similarity). 

9. Understand the meaning of percent and apply it in meaningful contexts. 



II. Measurement 

1. Select and use appropriate measurement instruments and units. 

2. Solve problems involving perimeter, area, volume, and surface area. 

3. Convert from one measurement to another within the same system 
(customary or metric). 

4. Apply concepts of accuracy of measurement in problem situations. 

5. Make and read scale drawings. 
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IIL Geometry and Spatial Sense 

1. Draw or sketch a figure given a verbal description or as 
needed to solve a problem. 

2. Given a figure, describe its geometric qualities. 

3. Predict results of combining, subdividing, and changing shapes. 

4. Identify the relationship (congruence, similarity) between a figure 
and its image under a transformation (informal: lines of symmetry, 
flips, turns, and slides). 

5. Transfer from three-dimensional to two-dimensional representations 
and vice versa. 

6. Classify figures in terms of congruence and similarity, and 
informally apply these relationships using proportional reasoning 
where appropriate. 

7. Use the Pythagorean relationship to solve problems. 

8. Represent problem situations with geometric models and apply properties 

of figures in meaningful contexts to solve mathematical and real-world problems. 

9. Measure and construct angles and other geometric figures. 



IV. Data Analysis, Statistics, and Probability 

1. Read, interpret, and make predictions using tables and graphs; 
interpolate or extrapolate from data. 

2. Organize and display data and use it to make inferences and decisions, 
using a variety of formats including circle graphs, scattergrams, stem-and-leaf 
plots, and box-and-whisker plots. 

3. Make generalizations based on sample results. 

4. Describe measures of central tendency and dispersion in real-world situations. 

5. Given certain situations and reported results, identify faulty arguments 
or misleading presentations of the data. 

6. Use basic concepts, trees, formulas for combinations and permutations, 
and other counting techniques to determine the number of ways an event 
can occur. 
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7. Use simulations to estimate probabilities and to make and describe 
predictions about expected outcomes. 

8. Use probability related to independent events. 

V. Algebra and Functions 

1. Extend or interpolate a pattern (complete missing terms). 

2. Translate patterns from one context to another. 

3. Understand and apply the concept of a variable. 

4. Use multiple representations for solving problems, including diagrams, 
models, and symbolic expressions. 

5. Apply a rectangular coordinate system, like cartesian coordinates, 
to the solution of problems. 

6. Represent and describe solutions to linear equations and inequalities 
to solve mathematical and real-world problems. 

7. Solve problems using algebraic expressions with methods such as 
substitutions, formulas, and equivalent forms. 

8. Describe how a change in one variable in a function results in a change 
in other variable(s) in the function (e.g., change in length of a rectangle 
produces a change in the area of the rectangle). 

VI. Problem Solving 

1. Use problem-solving approaches to investigate and understand 
mathematical content. 

2. Formulate problems from situations within and outside mathematics. 

3. Develop and apply a variety of strategies to solve problems, with 
emphasis on multistep and nonroutine problems. 

4. Verify and interpret results with respect to the original problem situation. 
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VIL Communication 



1 . Model situations using oral, written, concrete, pictorial, graphical 
and algebraic methods appropriate for the audience. 

2. Use the skills of reading, listening, and viewing to interpret and 
evaluate mathematical ideas. 

3. Understand and use mathematical definitions, notation, and vocabulary 
to explain mathematical ideas. 

VIIL Reasoning 

1. Recognize and apply various methods of reasoning, such as deductive 
and inductive reasoning, spatial reasoning, and reasoning with 
proportions and graphs. 

2. Make and evaluate mathematical conjectures and arguments. 



IX. Connections 

1 . Explain how different topics in mathematics are related. 

2. Solve problems and describe results using graphical, numerical, 
physical, algebraic, and verbal mathematical models or representations. 

3. Apply mathematical thinking and modeling to solve problems that 
arise in other disciplines, and in society. 



Total Average Score for Eighth-Grade Rigor 



H Twelfth Grade 

I. Number Sense, Properties, and Operations 

1. Model, order, round, and compute with real numbers. 

2. Determine the reasonableness and accuracy of answers to the 
required degree of accuracy. 
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IL Measurement 



1. Find volumes and surface area of geometric solids (e.g., cones, pyramids, 
prisms, and combined forms). 

2. Determine precision, accuracy, and error of measurement in problem situations. 



IIL Geometry and Spatial Sense 

1. Use transformations (translations, rotations, reflections, dilations, 
and symmetry) synthetically and algebraically. 

2. Apply similarity, congruence, and proportionality. 

3. Represent geometric figures and properties algebraically using 
coordinates and vectors. 

a. Use properties of lines (including distance, midpoint, slope, 
parallelism, and perpendicularity) to describe figures algebraically. 

b. Use vectors (including addition and subtraction) in problem situations. 



IV. Data Analysis, Statistics, and Probability 

1. Use measures of central tendency, correlation, dispersion, and shapes of 
distributions to describe statistical relationships and to make predictions 
and decisions. 

2. Use curve fitting to model and make predictions. 

3. Model real-world situations to determine the probabilities of dependent and 
independent events and compare these experimental probabilities with what 
would be expected based on theoretical models. 

4. Given a situation, describe a procedure for selecting an unbiased sample and 
identify the sources of sampling error. 

5. Design and implement a statistical experiment to study a problem and 
communicate the outcomes using appropriate statistics. 



V. Algebra and Functions 

1. Solve equations and inequalities algebraically, graphically, and using matrices. 

2. Represent and model relationships among quantities using symbols and 
expressions including graphs, matrices, sequences, series, and iterative relationships. 



ERIC 






40/CBE: Great Expectations? 



3. Solve quadratic equations using a variety of algebraic and graphical 
methods and tools. 

4. Use appropriate notation and terminology to describe functions and their 
properties, including domain and range. 

5. Compare and apply the numerical, symbolic, and graphical properties of a 
variety of functions and families of functions, examining general parameters 
and their effect on curve shape. 

6. Apply function concepts to model and deal with real-world situations. 

7. Use right triangle trigonometric functions to model real-world phenomena. 

VI. Problem Solving 

1. Select, modify, and/or create problem-solving approaches to investigate and 
understand mathematical content. 

2. Apply integrated mathematical problem-solving strategies to solve problems 
from within and outside mathematics. 

3. Apply the process of mathematical modeling to situations within and outside 
mathematics. 

VII. Communication 

1. Express mathematical ideas and generalizations orally and in writing, 
using appropriate vocabulary and notation. 

2. Read written presentations of mathematics with understanding. 

3. Ask clarifying and extending questions related to mathematics. 

VIII. Reasoning 

1. Make and test conjectures. 

2. Follow logical arguments and construct simple valid arguments. 

3. Judge the validity of arguments, formulating counterexamples as necessary. 
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IX. Connections 



1 . Recognize equivalent representations of the same concept and 
equivalent representations of procedures in different representations. 

2. Use the connections within mathematical topics and between 
mathematics and other disciplines. 



Total Average Score for Twelfth-Grade Rigor 



Scoring Summary and Final Grade 

Eighth-Grade Rigor Average Score 
Twelfth-Grade Rigor Average Score 



Final Average Score 



Grade for Rigor of Mathematics Standards 
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English Language Arts Framework 



I Fourth Grade 

L Read fluently and critically for a variety of purposes; 
synthesize information from a wide range of sources, including 
a variety of media. 

1. Read aloud independently with accuracy, fluency, and comprehension, 
using decoding and word attack skills (e.g., phonological awareness, phonic 
analysis, structural analysis) and comprehension strategies (e.g., inferencing; 
predicting, summarizing, and paraphrasing or retelling; generating questions; 
and identifying main ideas and details to support them). 

2. Distinguish between fact and opinion and main ideas and supporting 
details to draw meaning from various texts and media presentations. 

3. Read to perform a task and to gather information. 

4. Compare messages and organization in reading and media selections 
and relate content and message to previous knowledge. 

5. Identify basic visual media conventions, techniques, and processes used 

to support a variety of purposes (informational, inspirational, entertainment, 
manipulative); use media conventions and techniques to tell stories and 
communicate basic information. 



II. Read a variety of texts of different types (e.g., fiction, autobiography, 
historical narratives) by different authors and on different subjects. 

1. Read and comprehend the equivalent of at least thirty books per year in 
first through fourth grade of a specified variety in their entirety, both 
assigned in class and chosen for independent reading. 



III. Recognize and write in a variety of forms (including narratives, journals, 
stories, poetry, articles, instructions, analytical essays) to effectively convey 
intended meaning to particular audiences. 

1. Write for a specific purpose, audience, and context, using a simple 
organizing structure. 

2. Support a central idea with relevant facts and details in simple paragraph form. 

3. Recognize and employ the distinguishing features of different types of 
writing, such as instructions, narratives, journals, stories, poetry, drama, letters, 
news articles, and speeches. 




4 



D 



Score 
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4. Use key writing processes (with assistance), including prewriting, drafting, 
peer review, revising, editing, and publishing, to develop and express ideas. 



IV. Communicate effectively, adjusting use of spoken and visual language 
to convey intended meaning to a particular audience. 

1 . Demonstrate oral language skills including pace, volume, emphasis, 
pronunciation, audibility, and appropriate choice of words. 

2. Explain how verbal and nonverbal clues and variations in tone and gesture 
affect meaning. 

3. Determine meaning and intent in various forms of communication, including 
instructive, informative, persuasive, and entertaining, as well as in everyday life. 

4. Distinguish between fact and opinion, and main ideas and supporting 
details to draw meaning from discussions and oral presentations. 

5. Listen critically, ask clarifying questions, and express an opinion; support 
the opinion with specific evidence from the text or another relevant source. 

6. Plan and deliver oral presentations, matching purpose and message to 
audience, organizing content logically, and using visual aids, as appropriate. 



7. Generate and display ideas, information, and images. 

8. Identify strengths and weaknesses (based on established criteria) in one’s 

own and others’ presentations. 

V. Generate ideas and questions in order to develop research topics and methodologies; 
gather, evaluate, and synthesize data from a variety of sources; communicate 
discoveries in ways that suit specific purposes and audiences. 

1. Develop a research topic with teacher assistance. 

2. Locate relevant information from selected materials and begin to determine 

credibility with teacher assistance. 

3. Select materials and sources to match specific research purposes and explain 

the importance and usefulness of the selected materials. 

4. Gather information for specific purposes, with assistance, using traditional 
methods (e.g., library research, interviews, questionnaires) and current technology 

(e.g., library databases, CD-ROMs, video). 
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5. Organize information (e.g., take notes, outline) and target research to the 
specific needs of the project. 

6. Give precise credit for others’ ideas or information. 

7. Present a report that is a culmination of the research process. 



VL Write and speak clearly and with style, using the formal conventions of the 
English language, including grammar, spelling, punctuation, capitalization, 
sentence structure, word choice, paragraphing, and Hgurative language. 

1. Use correctly the basics of written and oral language, including grammar, 

sentence construction, paragraph structure, punctuation, capitalization, spelling, 

and usage in finished written and oral work. 

2. Explain and justify decisions about use of written language rules 

(e.g., capitalization, punctuation) in one’s own work and the work of others. 

3. Distinguish between situations and contexts in which standard English or 

the use of slang is appropriate; write and speak accordingly. 

4. Revise own work (based on established criteria) to improve 

quality and effectiveness. 

VIL Read a wide range of classic and contemporary literature’ from many places 
and periods and in many genres to build an understanding of the philosophical, 
ethical, and aesthetic^ dimensions of significant ideas and events. 

1. Identify basic figurative language and literary techniques (e.g., use of narration, 

dialogue, and characterization) in a variety of literary works including but not 

limited to fiction, nonfiction, and poetry. 

2. Explain how literary techniques (e.g., character, setting, plot, and conflict) 

are used in original fiction and non-fiction to create meaning. 

3. Identify the similarities and differences among selections of children’s literature 

and the ways in which these selections reflect the cultural background of the 

authors and the cultures in and about which they were written. 

4. Relate events, ideas, settings, and cultures of origin from one literary work 

to other texts and to their own lives. 

5. Use previewing strategies to select texts, media, and other material based on 

personal interests related to particular topics, authors, or genres. 



'In these standards we are defining literature as a wide-range of texts, including fiction, non-fiction, biography, essay, drama, 
poetry, lyrics, historical novels, and expository writing. 

^aesthetic: dealing with the perception and systematic study of beauty. 
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6. Review and recommend a variety of literature and media to others, articulating 
what is noteworthy about these selections. 

7. Identify similarities and differences of ideas and presentation in various 
media, and relate messages to previous knowledge and experience. 

Total Average Score for Fourth-Grade Rigor 



I Twelfth Grade 

I. Read fluently and critically for a variety of purposes; synthesize information 
from a wide range of sources, including a variety of media. 

1. Interpret and evaluate both literal and implied meaning in increasingly complex 
and varied reading materials, including: (a) texts which demonstrate sophisticated 
use of a variety of tools to convey meaning, (b) many texts on one topic or 

by one author to deepen knowledge, and (c) technical materials to develop 
understanding and expertise. _ 

2. Analyze texts from a variety of literary genres with regard to author’s craft 
(e.g., literary techniques, command of language); support opinions with 
evidence from a variety of sources besides the text (e.g., experience, 

news, other texts). \_ 

3. Interpret and critique modem media based on an analysis of its conventions, 

techniques, processes and purposes; apply the critique to produce effective 
communication in different media. \_ 



II. Read a variety of texts of different types (e.g., fiction, autobiography, 
historical narratives) by different authors and on different subjects. 

1 . Read and comprehend at least thirty books per year in ninth through twelfth 
grades of a specified variety in their entirety, both in class and independently. 

III. Recognize and write in a variety of forms (including narratives, journals, 
stories, poetry, articles, instructions, analytical essays) to effectively convey 
intended meaning to particular audiences. 

1. Write a narrative account (fictional or autobiographical), a persuasive essay, 
and a reflective essay, employing organizing structures, skills, and techniques 
appropriate to each. 

2. Demonstrate a sophisticated, personal writing style and different voices, 
employing both style and voice to write in a variety of genres and for a 
range of audiences. 
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IV. Communicate effectively, adjusting use of spoken and visual language to 
convey intended meaning to a particular audience. 

1. Evaluate literal and implied meaning in a variety of formal listening situations, 
including lectures, speeches, debates, dramatic presentations, and readings from 
literature and poetry. 

2. Modify message, delivery, and organization according to the verbal and 
nonverbal signals perceived during oral presentations and group discussions. 

3. Evaluate the quality and effectiveness of spoken and visual presentations; 
apply and provide constructive feedback (based on established criteria) to 
improve the quality and effectiveness of one’s own and others’ presentations. 

4. Generate effective artistic displays of complex ideas, information, and images. 



V. Gather, evaluate, and synthesize data from a variety of sources; communicate 
discoveries in ways that suit specific purposes and audiences. 

1 . Develop research designs and methods appropriate to the thesis and the 
requirements of the project; creatively pursue and locate information; 
adjust/redirect own theories and approaches based on information that is 
unexpected as well as that which is anticipated. 

2. Support an original thesis by conducting sustained research on a topic that 
synthesizes knowledge from more than one discipline and uses a variety of 
sources, such as technical journals and government publications. 

3. Evaluate the usefulness of information for a project in light of what one 
knows and one’s specific research needs. 

4. Give precise credit for information gained from conventional and 
unconventional sources (e.g., interviews and the World Wide Web) 

using an accepted documentation style (e.g.. Modem Languages Association, 

Chicago Manual of Style). 

5. Present research in formal written and oral forms to a designated audience ^ 

(e.g., peers and teachers; panel of experts; parents and general public). 
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VI. Write and speak clearly and with style, using the formal conventions of the English 
language, including grammar, spelling, punctuation, capitalization, sentence structure, 
word choice, paragraphing, and figurative language. 

1 . Independently, consistently, and with sophistication use the conventions of 

written and oral language, including grammar, sentence construction, paragraph 

structure, punctuation, spelling and usage. 



2. Analyze the development and changes in language over time. 

3. Revise own work and provide appropriate feedback to peers 

(based on established criteria) in order to improve quality and effectiveness. 



VII. Read a wide range of classic and contemporary literature from many places 
and periods and in many genres to build an understanding of the philosophical, 
ethical, and aesthetic dimensions of significant ideas and events. 

1. Analyze an author’s point of view, purpose, and historical and cultural context 

in relation to the text’s meaning. 

2. Analyze how written and spoken British literary works reflect the periods that 

shaped them (e.g., Anglo-Saxon, Medieval, Renaissance, the Enlightenment, 

Romantic, Victorian, Modem, and Contemporary). 

3. Analyze how written and spoken American literary works reflect the periods 
that shaped them (e.g., Undiscovered Country to Emerging Nation; the 
Revolutionary Period; National Growth; the New England Renaissance; the 

Division and the Civil War; the Frontier and Realism, Modem Period; 

Contemporary Period). 

4. Analyze how the predominant themes, genres, and styles of literature from the 

six inhabited continents reflect social and political activity as well as cultural 

aesthetic response to the world. 

5. Evaluate literary quality by analysis of elements such as author’s control 

of the storyteller’s voice, the author’s selections of significant details, and 

theme development. 

6. Develop and support a thesis about the craft and significance of particular 

written, spoken, and visual texts, both classic and contemporary, incorporating 

personal and communal criteria. 
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7. Analyze the conventions of a variety of literary genre (e.g., narrative structure 
in the novel; rhythm and line breaks in poetry; the use of lighting and music to 
develop mood in film) and the effects of those conventions on the reader or viewer. 

8. Compare and contrast literary texts to other works of the same time period 
(e.g., books, films, works of art, political speeches, comics) to understand 

the literary, artistic, political, and historical context out of which a text was created. [ 



Total Average Score for Twelfth-Grade Rigor 



Scoring Summary and Final Grade 

Fourth-Grade Rigor Average Score 
Twelfth-Grade Rigor Average Score 

Final Average Score 

Grade for Rigor of English Language Arts Standards 
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Scoring Rubrics for Mathematics and English Language Arts 



Rubric for Evaluating State Mathematics Standards 

4: To receive a four, state standards must require every student to learn all of the essential concepts and 
skills as defined by the framework benchmark at or above the level of sophistication specified. The 
state standards may capture different aspects of the framework benchmark as long as they still are 
indicative of similar concepts and skills at the same level of sophistication. 

For example, a framework benchmark for grade 12 states: 

Model real-world situations to determine the probabilities of dependent and independent 
events and compare these experimental probabilities with what would be expected 
based on theoretical models. 

State standards will be examined to determine whether they require mastery of all of the elements 
mentioned in the framework benchmark and at a comparable level of sophistication. In this case, the 
state standards must refer to or strongly suggest a study of experimental and theoretical probabilities 
of dependent and independent events at the sophistication level of “model [ing] real-world situations” 
and making comparisons. 

3: State standards will receive a three as they compare to a given framework benchmark if one of the fol- 
lowing three conditions is true: 

i. The state standards align with the framework benchmark, but are written at a lower level of 
sophistication. (For example. Determine the experimental and theoretical probabilities of an event) 

ii. The state standards are written at a level of sophistication that is comparable to the framework 
benchmark, but omit one of the essential concepts or skills or a few minor components. (For exaimple. 
Model real world situations to determine the probabilities of dependent events and compare these 
experimental probabilities with what would be expected based on theoretical models.) 

iii. The state standards omit an essential concept or skill and require a level of sophistication lower 
than the framework, but still convey high expectations. (For example. Compare the experimental 
probabilities of dependent events with what would be expected based on theoretical models.) 
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2: State standards will receive a two as they compare to a given framework benchmark if one of the 

following four conditions is true: 

i. The state standards align with the framework benchmark, but are written at a significantly lower 
level of sophistication. (For example, Explain the difference between a dependent and an inde- 
pendent event and between experimental and theoretical probabilities.) 

ii. The state standards are written at a level of sophistication that is comparable to the framework 
benchmark, but omit more than one of the essential concepts or skills. (For example. Model real- 
world situations to find the probability of a dependent event.) 

iii. The state standards omit essential concepts or skills and require a significantly lower level of 
sophistication, but still convey a relatively high level of sophistication. (For example. Calculate the 
probability of a given event occurring.) 

iv. The state standards seem to align implicitly with the framework benchmark, but the evaluator 
must make too many assumptions about the meaning of the standards to determine alignment. (For 
example. Determine the likelihood of an event occurring.) 

1: State standards will receive a one as they compare to a given framework benchmark if they could 
generally be said to address one of the essential concepts or skills in the framework benchmark but at 
an unacceptably low level of sophistication. (For example. Identify the common uses and misuses of 
probability and statistical analysis in the everyday world.) 

0: State standards will receive a zero if no standards comparable to the framework benchmark appear 
within the document. 
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Rubric for Evaluating State English Language Arts Standards 



4: To receive a four, state standards must require every student to learn all of the essential concepts or 
skills as defined by the framework benchmark at or above the level of sophistication specified. The 
state standards may capture different aspects of the framework benchmark as long as they still are 
indicative of similar concepts and skills at the same level of sophistication. 

For example, a framework benchmark for grade 12 states: 

Interpret and evaluate both literal and implied meaning in complex^ multiple, and varied reading 
materials, including: 

(a) texts which demonstrate sophisticated use of a variety of tools to convey meaning 

(b) many texts on one topic or by one author to deepen knowledge 

(c) technical materials to develop understanding and expertise 

State standards will be examined to determine whether they require mastery of all of the elements 
mentioned in the framework benchmark and at a comparable level of sophistication. In this case, the 
state standards must refer to or strongly suggest a study of literal and implied meaning in complex, 
multiple, and varied reading materials at the sophistication level of “interpret and evaluate.” The addi- 
tional information in the framework benchmark — types of texts (the statements following a, b, and c) — 
are for clarification purposes only and therefore affect the specificity, but not the rigor, of the benchmark. 

3: State standards will receive a three as they compare to a given framework benchmark if one of the 
following three conditions is true: 

i. The state standards align with the framework benchmark, but are written at a lower level of sophis- 
tication. (For example. Find examples of literal and implied meaning in a variety of texts.) 

ii. The state standards are written at a level of sophistication that is comparable to the framework 
benchmark, but omit one of the essential concepts or skills or a few minor components. (For exam 
pie. Interpret and evaluate meaning in complex, multiple, and varied reading materials.) 

iii. The state standards omit an essential concept or skill and require a level of sophistication lower than 
the framework, but still convey high expectations. (For example. Explain meaning in a variety of texts.) 
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2: State standards will receive a two as they compare to a given framework benchmark if one of the 
following four conditions is true: 

i. The state standards align with the framework benchmark, but are written at a significantly lower 
level of sophistication. (For example, Define literal and implied meaning in texts.) 

ii. The state standards are written at a level of sophistication that is comparable to the framework 
benchmark, but omit more than one of the essential concepts or skills. (For example, Interpret and 
evaluate literal meaning in a text.) 

iii. The state standards omit essential concepts or skills and require a significantly lower level of 
sophistication, but still convey a relatively high level of sophistication. (For example. Define 
implied meaning in texts.) 

iv. The state standards seem to align implicitly with the framework benchmark, but the evaluator must 
make too many assumptions about the meaning of the standards to determine alignment. (For 
example. Analyze the effects of complex literary devices (e.g., figurative language, irony) and com- 
plex elements (e.g., setting, plot) in a selection.) 

1 : State standards will receive a one as they compare to a given framework benchmark if they could gen- 
erally be said to address one of the essential concepts or skills in the framework benchmark but at an 
unacceptably low level of sophistication. (For example. Explain what a text means to you.) 

0: State standards will receive a zero if no standards comparable to the framework benchmark appear 
within the document. 
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About the Council for Basic Education 
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has historically cast itself as an independent, critical voice for education reform. In recent years, we have 
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better learning through better teaching. CBE periodicals, the monthly Basic Education and the quarterly 
Perspective, provide a regular platform for our advocacy, analysis, and criticism. Our readers nationwide 
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